--- title: Atmospheric Correction keywords: fastai sidebar: home_sidebar summary: "Out in the field, the OpenHSI camera measures light as it is reflected off surfaces. This light from the sun has absorption features from molecules/aerosols in the atmosphere and scattering. After converting the raw digital number counts into radiance" description: "Out in the field, the OpenHSI camera measures light as it is reflected off surfaces. This light from the sun has absorption features from molecules/aerosols in the atmosphere and scattering. After converting the raw digital number counts into radiance" nb_path: "03_atmos.ipynb" ---
{% raw %} 
/Users/eway/.pyenv/versions/3.8.3/lib/python3.8/site-packages/pandas/compat/__init__.py:97: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.
  warnings.warn(msg)
{% endraw %} {% raw %}
{% endraw %} {% raw %}
{% endraw %}

Atmospheric Profile from Balloon Sounding

Find the closest station at http://weather.uwyo.edu/upperair/sounding.html to use the atmospheric sounding on the day. find the station number and region code (for example, south pacific is pac and new zealand is nz) Default is Willis Island in Queensland, Australia. https://py6s.readthedocs.io/en/latest/helpers.html#importing-atmospheric-profiles-from-radiosonde-data

solar zenith angle already taken into account

Aerosol Profile from Aeronet

Permanently broken.

{% raw %}

class Model6SV[source]

Model6SV()

{% endraw %} {% raw %}
{% endraw %}

Py6S provides a method to loop over an array and return the 6SV result. However, it does not have a progress bar. For something that can take several minutes, not knowing how long it will take is genuinely frustrating. Therefore, we provide a modified version of SixSHelpers.Wavelengths.run_wavelengths called Model6SV.run_wavelengths that has a progress bar.

To use 6SV, Py6S will write input files and parse the output files. All this I/O is slow but we got to deal with it. Py6S is clever about it though and uses threading to do some compute while waiting for files. The modified version follows the same method except with the addition of a callback to update the progress bar when interations are done.

{% raw %} 
@patch
def _sixs_run_one_wavelength(self:Model6SV, wv:float) -> float:
    """Runs one instance of 6SV for one wavelength wv"""
    self.s.outputs = None
    a = copy.deepcopy(self.s)
    a.wavelength = Wavelength(wv)
    a.run()
    return SixSHelpers.Wavelengths.recursive_getattr(a.outputs, "pixel_radiance")
    
@patch
def run_wavelengths(self:Model6SV, wavelengths:np.array, n_threads:int = None) -> np.array:
    """Modified version of SixSHelpers.Wavelengths.run_wavelengths that has a progress bar.
    This implementation uses threading (through Python's multiprocessing API)."""
    from multiprocessing.dummy import Pool
    
    if n_threads is None: n_threads = num_cpus()
    with Pool(n_threads) as p, tqdm(total=len(wavelengths)) as pbar:
        res = [p.apply_async( self._sixs_run_one_wavelength, args=(wavelengths[i],), 
                callback=lambda _: pbar.update(1)) for i in range(len(wavelengths))]
        results = [r.get() for r in res]
    
    return np.array(results)
{% endraw %} {% raw %} 
model = Model6SV(λ_array=np.arange(0.35, 1., 0.01))
model.show()

100%|██████████| 65/65 [00:11<00:00,  5.73it/s]
{% endraw %} {% raw %} 
from scipy.interpolate import interp1d
rad_fit = interp1d(model.λ*1e3, model.radiance/10, kind='cubic')
#spec_rad_ref = np.float64(sfit(self.dcube.wavelength))
rad_fit(np.arange(400,800,10))

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-8-6f13be7a04a5> in <module>
      2 
      3 from scipy.interpolate import interp1d
----> 4 rad_fit = interp1d(model.λ*1e3, model.radiance/10, kind='cubic')
      5 #spec_rad_ref = np.float64(sfit(self.dcube.wavelength))
      6 rad_fit(np.arange(400,800,10))

AttributeError: 'Model6SV' object has no attribute 'λ'
{% endraw %} {% raw %} 
cam_prop = CameraProperties("assets/cam_settings.txt","assets/cam_calibration.pkl")

cam_prop

settings = 
{'camera_id': '0', 'row_slice': [197, 649], 'index2wavelength_range': [397.35239556600595, 823.5444046337864, 0.2085273916954825], 'resolution': [772, 2064], 'fwhm_nm': 4, 'exposure_ms': 10, 'longitude': -17.7, 'latitude': 146.1, 'datetime_str': '2021-05-26 03:26', 'altitude': 0.12, 'radiosonde_station_num': 94299, 'radiosonde_region': 'pac', 'sixs_path': 'assets/6SV1.1/sixsV1.1', 'luminance': 30000}

calibration = 
{'camera_id': '0', 'HgAr_pic': array([[32, 30, 30, ..., 31, 31, 32],
       [32, 32, 34, ..., 33, 32, 32],
       [32, 31, 30, ..., 32, 32, 31],
       ...,
       [31, 31, 31, ..., 33, 31, 33],
       [29, 32, 31, ..., 31, 31, 32],
       [32, 32, 32, ..., 31, 32, 33]], dtype=uint32), 'flat_field_pic': array([[33, 34, 33, ..., 34, 36, 33],
       [34, 33, 36, ..., 33, 33, 35],
       [33, 33, 32, ..., 33, 35, 32],
       ...,
       [33, 32, 34, ..., 36, 32, 37],
       [35, 33, 34, ..., 34, 33, 33],
       [33, 33, 33, ..., 31, 33, 36]], dtype=uint32), 'smile_shifts': array([ 9,  9, 10, 10, 10, 10, 10, 10,  9,  9,  9,  9,  9,  9,  9,  9,  9,
        9,  9,  9,  9,  9,  9,  9,  9,  9,  9,  9,  9,  9,  9,  9,  9,  9,
        9,  9,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,
        8,  8,  8,  8,  8,  8,  8,  7,  8,  8,  8,  7,  7,  7,  7,  7,  7,
        7,  7,  7,  7,  7,  7,  7,  7,  7,  7,  7,  7,  7,  7,  7,  7,  7,
        7,  7,  7,  7,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,
        6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  5,  5,
        5,  5,  5,  5,  5,  5,  5,  5,  5,  5,  5,  5,  5,  5,  5,  5,  5,
        5,  5,  5,  5,  5,  5,  5,  5,  5,  5,  5,  4,  4,  4,  4,  4,  4,
        4,  4,  4,  4,  4,  4,  4,  4,  4,  4,  4,  4,  4,  4,  4,  4,  4,
        4,  4,  4,  4,  4,  4,  4,  4,  3,  3,  3,  3,  3,  3,  3,  3,  3,
        3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,
        3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,
        3,  3,  3,  3,  3,  2,  3,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,
        2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,
        2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,
        2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,
        2,  2,  1,  1,  1,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,
        2,  1,  2,  2,  2,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
        1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
        1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
        1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
        1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
        1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,
        1,  1,  1,  1,  1,  1,  1,  1,  1,  0,  0,  0,  1,  0,  1,  1,  0,
        0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
        0,  0,  0,  0,  0,  0,  0,  0,  0,  0], dtype=int16), 'rad_ref': <xarray.DataArray 'datacube' (luminance: 13, exposure: 9, cross_track: 452, wavelength: 108)>
array([[[[    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         ...,
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0]],

        [[    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         ...,
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0]],

        [[    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         ...,
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0]],

        ...,

        [[    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         ...,
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0]],

        [[    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         ...,
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0]],

        [[    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         ...,
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0],
         [    0,     0,     0, ...,     0,     0,     0]]],


       [[[  634,   643,   652, ...,   630,   631,   631],
         [  636,   642,   648, ...,   640,   642,   643],
         [  635,   642,   648, ...,   847,   857,   857],
         ...,
         [  634,   640,   647, ...,  1072,  1051,  1031],
         [  633,   640,   647, ...,  1072,  1048,  1025],
         [  635,   642,   647, ...,  1067,  1042,  1021]],

        [[  639,   651,   659, ...,   636,   636,   635],
         [  641,   649,   656, ...,   647,   648,   650],
         [  641,   650,   657, ...,   899,   909,   908],
         ...,
         [  641,   649,   656, ...,  1171,  1148,  1120],
         [  639,   647,   656, ...,  1169,  1143,  1112],
         [  639,   650,   655, ...,  1162,  1133,  1110]],

        [[  645,   657,   668, ...,   639,   640,   640],
         [  645,   656,   664, ...,   651,   654,   656],
         [  647,   657,   666, ...,   951,   958,   961],
         ...,
         [  647,   656,   663, ...,  1267,  1237,  1209],
         [  645,   655,   663, ...,  1267,  1235,  1203],
         [  646,   657,   665, ...,  1259,  1221,  1197]],

        ...,

        [[  674,   694,   709, ...,   663,   664,   664],
         [  675,   693,   707, ...,   684,   686,   689],
         [  675,   692,   710, ...,  1208,  1221,  1222],
         ...,
         [  675,   690,   707, ...,  1750,  1699,  1651],
         [  673,   689,   703, ...,  1747,  1697,  1641],
         [  674,   692,   706, ...,  1732,  1674,  1629]],

        [[  672,   696,   718, ...,   659,   660,   659],
         [  672,   696,   715, ...,   684,   686,   691],
         [  674,   696,   717, ...,  1341,  1357,  1357],
         ...,
         [  676,   693,   714, ...,  2020,  1963,  1893],
         [  673,   695,   712, ...,  2021,  1957,  1888],
         [  674,   695,   715, ...,  1999,  1934,  1875]],

        [[  700,   730,   758, ...,   681,   682,   681],
         [  702,   730,   759, ...,   715,   719,   722],
         [  702,   732,   759, ...,  1598,  1612,  1613],
         ...,
         [  705,   730,   755, ...,  2513,  2427,  2343],
         [  701,   729,   753, ...,  2503,  2413,  2328],
         [  703,   731,   758, ...,  2472,  2383,  2310]]],


       [[[  675,   695,   714, ...,   661,   662,   662],
         [  676,   694,   712, ...,   682,   684,   688],
         [  676,   694,   713, ...,  1201,  1213,  1215],
         ...,
         [  677,   695,   711, ...,  1766,  1715,  1665],
         [  674,   693,   709, ...,  1764,  1711,  1657],
         [  678,   696,   711, ...,  1747,  1694,  1646]],

        [[  689,   712,   735, ...,   673,   674,   672],
         [  689,   713,   734, ...,   698,   697,   702],
         [  691,   713,   734, ...,  1327,  1340,  1342],
         ...,
         [  692,   710,   733, ...,  2009,  1951,  1886],
         [  689,   711,   730, ...,  2004,  1944,  1881],
         [  692,   713,   734, ...,  1985,  1921,  1866]],

        [[  705,   730,   757, ...,   684,   684,   683],
         [  705,   730,   755, ...,   712,   712,   719],
         [  703,   731,   756, ...,  1450,  1466,  1466],
         ...,
         [  706,   730,   753, ...,  2248,  2177,  2103],
         [  703,   728,   750, ...,  2245,  2172,  2097],
         [  704,   729,   753, ...,  2219,  2144,  2083]],

        ...,

        [[  754,   800,   846, ...,   721,   722,   720],
         [  756,   800,   846, ...,   769,   771,   780],
         [  756,   802,   847, ...,  2048,  2077,  2079],
         ...,
         [  762,   802,   844, ...,  3427,  3304,  3177],
         [  757,   801,   842, ...,  3423,  3301,  3172],
         [  759,   802,   848, ...,  3377,  3255,  3140]],

        [[  815,   873,   929, ...,   772,   772,   772],
         [  816,   873,   929, ...,   833,   838,   846],
         [  818,   875,   931, ...,  2435,  2467,  2476],
         ...,
         [  824,   877,   926, ...,  4167,  4016,  3860],
         [  818,   875,   925, ...,  4146,  3996,  3844],
         [  818,   876,   928, ...,  4105,  3945,  3802]],

        [[  886,   959,  1037, ...,   827,   830,   826],
         [  887,   959,  1033, ...,   910,   917,   924],
         [  890,   965,  1040, ...,  3046,  3089,  3102],
         ...,
         [  894,   966,  1039, ...,  5367,  5166,  4944],
         [  891,   964,  1034, ...,  5344,  5147,  4941],
         [  891,   965,  1037, ...,  5280,  5076,  4880]]],


       ...,


       [[[ 1256,  1455,  1665, ...,  1136,  1134,  1130],
         [ 1250,  1454,  1664, ...,  1631,  1672,  1719],
         [ 1266,  1465,  1677, ...,  8453,  8388,  8278],
         ...,
         [ 1289,  1485,  1684, ..., 11375, 10883, 10387],
         [ 1281,  1480,  1676, ..., 11299, 10849, 10367],
         [ 1277,  1478,  1681, ..., 11172, 10719, 10247]],

        [[ 1384,  1625,  1878, ...,  1240,  1242,  1237],
         [ 1383,  1625,  1878, ...,  1845,  1905,  1956],
         [ 1398,  1635,  1895, ..., 10098, 10027,  9877],
         ...,
         [ 1427,  1656,  1904, ..., 13615, 13037, 12443],
         [ 1416,  1659,  1897, ..., 13539, 12982, 12398],
         [ 1411,  1648,  1898, ..., 13371, 12818, 12253]],

        [[ 1514,  1795,  2094, ...,  1349,  1352,  1342],
         [ 1516,  1795,  2095, ...,  2072,  2137,  2201],
         [ 1532,  1815,  2113, ..., 11763, 11677, 11493],
         ...,
         [ 1561,  1840,  2122, ..., 15835, 15165, 14456],
         [ 1551,  1828,  2111, ..., 15744, 15096, 14428],
         [ 1551,  1830,  2114, ..., 15565, 14911, 14263]],

        ...,

        [[ 2173,  2645,  3167, ...,  1890,  1895,  1878],
         [ 2171,  2650,  3162, ...,  3178,  3299,  3410],
         [ 2206,  2688,  3193, ..., 20087, 19940, 19583],
         ...,
         [ 2255,  2730,  3213, ..., 27052, 25897, 24684],
         [ 2239,  2716,  3187, ..., 26888, 25788, 24605],
         [ 2229,  2701,  3198, ..., 26573, 25451, 24327]],

        [[ 2540,  3141,  3784, ...,  2204,  2202,  2185],
         [ 2540,  3146,  3784, ...,  3822,  3966,  4105],
         [ 2581,  3189,  3829, ..., 25052, 24869, 24404],
         ...,
         [ 2642,  3246,  3854, ..., 33746, 32304, 30756],
         [ 2627,  3226,  3822, ..., 33531, 32155, 30688],
         [ 2618,  3209,  3822, ..., 33171, 31786, 30335]],

        [[ 3197,  3990,  4855, ...,  2745,  2745,  2732],
         [ 3196,  3993,  4868, ...,  4929,  5130,  5311],
         [ 3254,  4052,  4913, ..., 33405, 33127, 32527],
         ...,
         [ 3332,  4126,  4946, ..., 44948, 43032, 40924],
         [ 3310,  4100,  4908, ..., 44677, 42827, 40861],
         [ 3300,  4086,  4905, ..., 44152, 42297, 40396]]],


       [[[ 1274,  1486,  1710, ...,  1198,  1203,  1193],
         [ 1274,  1485,  1707, ...,  2650,  2771,  2892],
         [ 1286,  1505,  1725, ..., 10606, 10364, 10031],
         ...,
         [ 1309,  1516,  1731, ..., 11979, 11464, 10934],
         [ 1302,  1516,  1722, ..., 11860, 11374, 10884],
         [ 1299,  1507,  1724, ..., 11778, 11286, 10799]],

        [[ 1408,  1657,  1932, ...,  1318,  1321,  1316],
         [ 1412,  1663,  1933, ...,  3076,  3217,  3358],
         [ 1428,  1684,  1960, ..., 12679, 12391, 11986],
         ...,
         [ 1455,  1704,  1960, ..., 14322, 13717, 13090],
         [ 1446,  1696,  1949, ..., 14186, 13629, 13012],
         [ 1442,  1694,  1954, ..., 14118, 13514, 12949]],

        [[ 1548,  1837,  2157, ...,  1437,  1448,  1439],
         [ 1546,  1843,  2153, ...,  3522,  3681,  3861],
         [ 1564,  1863,  2179, ..., 14776, 14427, 13949],
         ...,
         [ 1594,  1888,  2185, ..., 16692, 15974, 15225],
         [ 1583,  1880,  2175, ..., 16524, 15851, 15137],
         [ 1578,  1876,  2177, ..., 16414, 15734, 15045]],

        ...,

        [[ 2201,  2700,  3244, ...,  2036,  2044,  2031],
         [ 2211,  2709,  3256, ...,  5703,  5976,  6273],
         [ 2244,  2751,  3296, ..., 25196, 24613, 23755],
         ...,
         [ 2289,  2790,  3308, ..., 28509, 27254, 25982],
         [ 2278,  2780,  3280, ..., 28224, 27078, 25846],
         [ 2266,  2765,  3285, ..., 28041, 26854, 25663]],

        [[ 2611,  3231,  3921, ...,  2397,  2410,  2402],
         [ 2611,  3247,  3926, ...,  6998,  7352,  7718],
         [ 2660,  3288,  3974, ..., 31439, 30713, 29651],
         ...,
         [ 2716,  3342,  3986, ..., 35596, 34055, 32438],
         [ 2695,  3323,  3954, ..., 35260, 33792, 32276],
         [ 2687,  3307,  3960, ..., 35023, 33521, 32036]],

        [[ 3297,  4123,  5031, ...,  3012,  3022,  3010],
         [ 3300,  4142,  5042, ...,  9153,  9600, 10092],
         [ 3354,  4198,  5108, ..., 41824, 40861, 39439],
         ...,
         [ 3432,  4266,  5133, ..., 47402, 45369, 43180],
         [ 3408,  4245,  5090, ..., 46984, 45022, 42994],
         [ 3393,  4218,  5089, ..., 46585, 44634, 42654]]],


       [[[  610,   612,   612, ...,   612,   611,   613],
         [  612,   612,   610, ...,   612,   612,   613],
         [  612,   611,   611, ...,   609,   612,   611],
         ...,
         [  612,   611,   611, ...,   612,   613,   611],
         [  611,   611,   611, ...,   613,   612,   607],
         [  613,   613,   611, ...,   614,   612,   612]],

        [[  610,   612,   612, ...,   612,   611,   612],
         [  612,   611,   609, ...,   614,   614,   615],
         [  612,   611,   611, ...,   610,   613,   612],
         ...,
         [  612,   611,   611, ...,   612,   614,   612],
         [  611,   610,   612, ...,   614,   614,   608],
         [  614,   614,   612, ...,   615,   613,   613]],

        [[  610,   612,   612, ...,   612,   610,   612],
         [  614,   612,   611, ...,   613,   615,   615],
         [  612,   612,   611, ...,   609,   613,   613],
         ...,
         [  612,   612,   611, ...,   613,   614,   612],
         [  614,   612,   612, ...,   615,   614,   609],
         [  614,   613,   612, ...,   615,   613,   612]],

        ...,

        [[  595,   596,   596, ...,   596,   595,   596],
         [  595,   596,   595, ...,   597,   597,   598],
         [  596,   596,   594, ...,   595,   597,   598],
         ...,
         [  596,   595,   596, ...,   596,   599,   596],
         [  596,   595,   596, ...,   598,   597,   591],
         [  598,   598,   596, ...,   600,   598,   597]],

        [[  595,   598,   597, ...,   596,   595,   597],
         [  597,   597,   596, ...,   599,   599,   600],
         [  597,   598,   596, ...,   594,   598,   597],
         ...,
         [  598,   596,   596, ...,   597,   600,   597],
         [  597,   597,   597, ...,   599,   600,   593],
         [  597,   598,   597, ...,   601,   597,   597]],

        [[  597,   600,   599, ...,   599,   599,   600],
         [  598,   598,   596, ...,   599,   599,   600],
         [  597,   599,   596, ...,   595,   600,   598],
         ...,
         [  600,   597,   599, ...,   598,   602,   599],
         [  599,   599,   598, ...,   602,   601,   596],
         [  599,   599,   597, ...,   602,   599,   598]]]], dtype=int32)
Coordinates:
  * cross_track  (cross_track) int32 0 1 2 3 4 5 6 ... 446 447 448 449 450 451
  * wavelength   (wavelength) float32 397.3 401.3 405.2 ... 813.3 817.3 821.3
  * luminance    (luminance) int32 0 2500 5000 7500 ... 45000 47500 50000 0
  * exposure     (exposure) int32 5 6 7 8 9 10 12 15 20, 'sfit': <scipy.interpolate.interpolate.interp1d object at 0x12ed98ef0>, 'rad_fit': <scipy.interpolate.interpolate.interp1d object at 0x12eef2360>}
{% endraw %}